Characteristics , Impact , and Tolerance of Partial Disk Failures

Hard-disk failures are one of the primary causes of data loss in both enterprise storage systems and personal computers. Most disk failures are partial failures, where only some sectors are unavailable due to a latent sector error or some blocks are silently corrupted. This dissertation focuses on all aspects of such partial disk failures – their characteristics, their impact on different systems, and techniques that can be used tolerate them. We perform the first large-scale study of partial disk failures, involving 1.53 million disks in more than 50,000 storage systems. We find that partial disk failures affect a large percentage of disks (e.g., in the worst case, latent sector errors affect up to 20% of the disks in 2 years). We also find that (i) inexpensive SATA drives have a higher probability of developing partial disk failures, (ii) failures are not independent; failures within the same disk have high spatial and temporal locality, and (iii) many failures are detected by background scans of disk blocks called “disk scrubbing.” We examine the impact of partial disk failures on a variety of systems. We use model checking to examine data protection in RAID systems. We find that schemes in many RAID systems are broken; they do not protect against one or more failures, leading to unrecoverable data loss or corrupt data being returned to applications. We apply type-aware fault injection to examine the impact of partial disk failures on the virtual-memory systems of Linux, FreeBSD, and Windows XP. We find that these systems use simplistic or inconsistent failure-handling policies, thus causing data corruption and system-security violations. We analyze the impact of corrupt on-disk pointers on two file systems, NTFS and ext3. We find that these systems do not use available fault-tolerance techniques effectively, resulting in data loss and non-mountable file systems. Overall, we find that a single system cannot be depended upon to reliably store data.